Looking at the polls
27/01/16 05:25
In this season of polls and politicians who seem hang on every opinion survey taken, I need to say that I am a skeptic when it comes to the value of polls as measures of popular opinion. It has been well documented that the wording of questions and how they are asked can have dramatic effects on poll results. When it comes to political polls, he pollsters are well-educated in what kinds of results are wanted and are strategic in getting their polls to reach pre-determined conclusions. If a candidate is willing to pay enough money, a pollster can come up with a survey that gives the results the candidate wants.
This is hardly scientifically accurate research.
Having said that, I don’t ignore the polls entirely. For many years the results of studies by the Pew Research Center have provided statistics on religion that I have not only read, but even quoted in speaking about trends in demographics and their effects on the practice of religion in America. Centerpiece of the Pew Charitable Trust’s initiative to increase citizen involvement in community issues, the Pew Research Center is nonpartisan and appears to be genuinely interested in the effects of religion on everyday life.
A recent survey by Pew, however, has left me scratching my head. On the surface, it seems to make sense to figure out the relationship between science and religion. The results of the poll roughly follow what I might have expected. While a majority of those studied (59%) believe that religion and science are in conflict, a much smaller percentage (30%) find conflicts between their personal faith and science. Perhaps more interesting is the finding that the people who are most likely to stay that religion and science are in conflict are people who attend religious services seldom or never. In other words, those who are least involved see the conflict as being more intense than those who regularly practice their faith.
Upon a deeper look at the poll, however, I am convinced that this particular poll gives us very little information that is useful to either scientists or religious leaders.
As they say, “to get good answers, you have to ask good questions.” The pew question, “Generally, do you think science and religion are often in conflict?” is not a good question. It is vague. What does it mean by science? Is it referring to specific theories or the practice of science in general? And does Religion refer to the beliefs of evangelical Protestants, secular Jews, Buddhists, or some other specific set of beliefs? Or does religion refer to the institutions of religion and the practices of the faithful? The question is not at all clear. That question, however, is at the heart of the Pew Research Center’s study.
My worry about poor polls is that rather than study public opinion, the polls themselves can play a role in forming public opinion. Instead of being objective, they can contribute to misinformation. The Pew Study, in my opinion subscribes to a very narrow view of religion and of science. This is compounded by a lack of educated reporters who seem to be unduly influenced by anything that claims to report truth. By practicing poor science, the study itself contributes to unnecessary conflict between science and religion.
Let’s look at what we do and what we do not know about the poll. Pew surveyed 2,000 Americans about their perceptions of the relationship between religion and science. They claim that their sample reflects that religious and cultural diversity of the nation, though what criteria was imposed to insure that diversity is not clear. Were the levels of involvement in religion of those polled similar to the general public? Should we turn to the general public for information about the relationship between religion and science? Would we not get more accurate information if the poll paid attention to the relative expertise of those questioned? How well are religious minorities represented in the poll?
The study was conducted over the phone using a list of simple multiple choice questions. Multiple choice questions aren’t the most useful ways to delve into theological understanding. People’s feelings and opinions about religion often shift greatly. They might believe one thing while they are sitting in church and another thing when they are in the midst of a business meeting. The study also assumes that people have pre-formed opinions on religious issues. Those who do not regularly participate in church may not have spent much time thinking about religious issues.
Consider this question, that was asked as part of the study: “Thinking about the use of biological engineering to create artificial organs for humans needing a transplant operation, would you say this is making appropriate use of medical advances OR is it taking medical advances too far?” An honest answer to the question might require nuances not available in the either/or choice. Furthermore, most people would need some time to reflect and think before arriving at an answer. information gained in a short telephone conversation would be, at best, incomplete.
Poll respondents were asked about five such complex questions every minute. Those polled are receiving questions so rapidly that they often have no idea whether they really give the right answer.
Another significant problem with the poll is what it does not tell us. We do not know the response rate. In getting 2,000 respondents, how many people simply refused to participate? What percentage of people called agreed to participate in the study in the first place. At our home, we almost always refuse to participate in telephone surveys. We have valid and important opinions, but we don’t hold much stock in that type of research. It remains unknown whether or not the subset of people who are willing to participate in telephone surveys are in anyway representative of the general population.
So I read the surveys with a combination of interest and skepticism. On the other hand, I read the reports about such surveys with less interest and my skepticism reaches close to cynicism when it comes to the media reports.
It requires more than a few telephone calls to arrive at the truth.
This is hardly scientifically accurate research.
Having said that, I don’t ignore the polls entirely. For many years the results of studies by the Pew Research Center have provided statistics on religion that I have not only read, but even quoted in speaking about trends in demographics and their effects on the practice of religion in America. Centerpiece of the Pew Charitable Trust’s initiative to increase citizen involvement in community issues, the Pew Research Center is nonpartisan and appears to be genuinely interested in the effects of religion on everyday life.
A recent survey by Pew, however, has left me scratching my head. On the surface, it seems to make sense to figure out the relationship between science and religion. The results of the poll roughly follow what I might have expected. While a majority of those studied (59%) believe that religion and science are in conflict, a much smaller percentage (30%) find conflicts between their personal faith and science. Perhaps more interesting is the finding that the people who are most likely to stay that religion and science are in conflict are people who attend religious services seldom or never. In other words, those who are least involved see the conflict as being more intense than those who regularly practice their faith.
Upon a deeper look at the poll, however, I am convinced that this particular poll gives us very little information that is useful to either scientists or religious leaders.
As they say, “to get good answers, you have to ask good questions.” The pew question, “Generally, do you think science and religion are often in conflict?” is not a good question. It is vague. What does it mean by science? Is it referring to specific theories or the practice of science in general? And does Religion refer to the beliefs of evangelical Protestants, secular Jews, Buddhists, or some other specific set of beliefs? Or does religion refer to the institutions of religion and the practices of the faithful? The question is not at all clear. That question, however, is at the heart of the Pew Research Center’s study.
My worry about poor polls is that rather than study public opinion, the polls themselves can play a role in forming public opinion. Instead of being objective, they can contribute to misinformation. The Pew Study, in my opinion subscribes to a very narrow view of religion and of science. This is compounded by a lack of educated reporters who seem to be unduly influenced by anything that claims to report truth. By practicing poor science, the study itself contributes to unnecessary conflict between science and religion.
Let’s look at what we do and what we do not know about the poll. Pew surveyed 2,000 Americans about their perceptions of the relationship between religion and science. They claim that their sample reflects that religious and cultural diversity of the nation, though what criteria was imposed to insure that diversity is not clear. Were the levels of involvement in religion of those polled similar to the general public? Should we turn to the general public for information about the relationship between religion and science? Would we not get more accurate information if the poll paid attention to the relative expertise of those questioned? How well are religious minorities represented in the poll?
The study was conducted over the phone using a list of simple multiple choice questions. Multiple choice questions aren’t the most useful ways to delve into theological understanding. People’s feelings and opinions about religion often shift greatly. They might believe one thing while they are sitting in church and another thing when they are in the midst of a business meeting. The study also assumes that people have pre-formed opinions on religious issues. Those who do not regularly participate in church may not have spent much time thinking about religious issues.
Consider this question, that was asked as part of the study: “Thinking about the use of biological engineering to create artificial organs for humans needing a transplant operation, would you say this is making appropriate use of medical advances OR is it taking medical advances too far?” An honest answer to the question might require nuances not available in the either/or choice. Furthermore, most people would need some time to reflect and think before arriving at an answer. information gained in a short telephone conversation would be, at best, incomplete.
Poll respondents were asked about five such complex questions every minute. Those polled are receiving questions so rapidly that they often have no idea whether they really give the right answer.
Another significant problem with the poll is what it does not tell us. We do not know the response rate. In getting 2,000 respondents, how many people simply refused to participate? What percentage of people called agreed to participate in the study in the first place. At our home, we almost always refuse to participate in telephone surveys. We have valid and important opinions, but we don’t hold much stock in that type of research. It remains unknown whether or not the subset of people who are willing to participate in telephone surveys are in anyway representative of the general population.
So I read the surveys with a combination of interest and skepticism. On the other hand, I read the reports about such surveys with less interest and my skepticism reaches close to cynicism when it comes to the media reports.
It requires more than a few telephone calls to arrive at the truth.